PRET A Reporter: evaluating Digital Libraries alone and in context

There is a multiplicity of methods available for assessing the usability and use of any information retrieval (IR) system: some empirical, some relying more on expert review; some appropriate to questions about the details of a particular interface design, others better suited to understanding the ways such systems impact on working practices. In this paper, we present the PRET A Reporter framework for structuring user-centred evaluation studies, illustrating its use with examples of studies conducted on one particular class of IR system: digital libraries. These examples are drawn from various research projects conducted over six years. They illustrate the importance of understanding the evaluation problem, working with available resources and constraints, considering ethics in planning and reporting studies, selecting suitable techniques for data collection and analysis and reporting results in different forms to different audiences.